Issues Identified Before and During 2004.04.20 CIMT/SSDS Meeting

1. Q: One of the items is to pass data to/from the SSDS. Does that mean that we (CIMT) are responsible for reproducing the same types of plots and such that MBARI is already creating internally? Or is it possible to combine these two processes?
   1. A: Both processes can be combined. The "passing data from and to" is to enable automated non-SSDS processing.
2. #13 on the prioritization list should be bumped up a bit...if we can't get through the firewall to see the data we are going to get negative feedback from NOAA.
   1. Understood. Is in fact in progress.
3. Are we planning on a watch circle type plot for GPS?
   1. Yes (though note it is in the custom plots category, Priority #16).
4. Is tic mark beginning of day?
   1. Yes, midnight on that day. These formats can be changed.
5. How is development affected by selection of particular plotting package (e.g., JFreeChart) to support SSDS requirements?
   1. It affects how our internal services work (e.g., the callable interface we're using for this demonstration), but users are expected to download data (via TBD interfaces) to work with more advanced applications.
6. How will user know what URL to specify to get what they want?
   1. At first by asking us, but soon enough by filling out a query form (which will then generate a URL, and the URL can be modified and reused).
7. Concern about query page being time-oriented. Can't we make it easy to do this by region too (i.e., lat/lon/depth)? You do need to consider this for CIMT, but a simple (nominal) concept is probably all we should do at first. It doesn't have to be embedded in data, but incorporated in header of a plot. SSDS should eventually be able to query on these 4 parameters: time, lat, lon, depth.
   1. Points well taken. Please note this is hard, and not on the priority list.
8. How do you envision getting data from this system on a recurring basis? This could be used, for example, to connect two data systems together. This is Likely to be common.
   1. We have to design this feature (see Priorities #5 and #8).
9. Would be nice to be able to give feedback about the data, so that the feedback lives with the data.
   1. This is a significant operational challenge across the board at MBARI. But, the architecture we have could be tuned to such a thing. (There are 3 distinct parts to this request; routine automated QC flagging; injecting comments during post-processing, manually or automatically; and user feedback on existing data. The last might be accomplished by publishing the "instrument owner" with the plots.) Note this is not on the priority list.
10. Why can't we query more/better/sooner? How about directly querying the database? Point and click query access can wait, but developer queries must be supported soon. (When will they be available?)
    1. An interface which can support at least some developer queries will be available sometime 'soon' (at its most basic possibly within a few weeks). That kind of interface is implied in Priorities #3, #5, #8, #11, #12, and #16. The query documented by Priority #14 is the point-and-click type.
11. Processing data internally, by moving currently external processing into SSDS, is desirable (e.g., for Metsys and other Bahr data).
    1. We are evaluating all the external processing requirements before proceesing on Priority #8, but my (John G) first approach is to work with existing interfaces as much as possible, for the sake of speed.
12. Add lookup by platform name to device table front end.
    1. Good idea. Have it mind, but not on the priority list.
13. Not clear how to get to a specific data variables/data sets. An interface to 'query for data' is needed, but it isn't on the priority list. (How to describe this need, in the context of the priority list, isn't clear.)
    1. We will be better able to respond to this, and consider what priority it should have, at the next User Story meeting.
14. What time frame are you likely to have some of this work done – do you even know?
    1. We have a decent idea. We expect to be one more step toward a final solution for #8 and #9 by the next User Story meeting on 5/6/04, with a prototype to review. Also at that meeting, we may have significant progress on one or all of the following (current status in parentheses):
       #10: (solution in mind, implementation in progress)
       #11: (prototype demonstrated today, wider access intended by 5/6)
       #12: (notional solution in mind)
       #13: (PC in house, final architecture still undetermined)
       #16: (none currently developed, but this wouldn't be hard, especially with a collaborator)
       Status of infrastructure items:
       #1: Done for many cases, some cases still in progress.
       #2: In progress, 5 MTM instruments described.
       #3: Example demonstrated today, some enhancements needed for end users, and still needs to be "released" and possibly put service outside firewall.
       #4: Coding can now begin once agreed SIAM metadata interface is documented.
       #5: Prototype developed, considerable standardization remains.
       #6: In research.
       #7: No activity.

Identified/Prioritized User Stories (with caveats).

Items 1-7 are infrastructure services. Most will serve any ingested and described data. Items 8 and beyond are specific CIMT implementations/utilizations of the infrastructure services.

Rough scope is indicated in brackets: A=5 work days, B=5-10 work days, C=10-20 work days.

Done

• Ingest raw SIAM data packets.
• Manage metadata for data packets (basic infrastructure; minor mods needed for CIMT).
• User accesses raw data packets via web (in place for MTM2; tuning needed for CIMT).
• Prototype tool available to confirm consistency of data.

Infrastructure

• Parse ASCII streams A
• Define metadata describing data B
• Create general web plotting capability (to support QC-style, parameter vs time plots only) C
• Simplify SIAM data ingest (internal, no visible behavior change) B
• Add Data notification mechanism or mechanisms (to maximize # of external processes served data) B
• Add data submission mechanism or mechanisms (to maximize # of external processes submitting data) B
• Create a general query interface (expect only one or two basic queries supported for CIMT; see #15 below) B

CIMT-Specific

• Provide data to specific external processes (TBD which ones) B
• Accept data from specific external processes (TBD which ones) A
• Ingest SOON data as a data stream B
• Provide basic plots of raw data (note more advanced plots were deemed not critical by deployment) A
• Merge items according to timestamp, optionally at a given time interval. (Focused on data.)B

Not sure

The following 4 items are on the bubble'; ideally all 4 can be accomplished, but possibly none of them can, depending on speed of the first 12 items. (I expect 2 of these can be done.)

• Make plots available outsisde the firewall (i..e, publicly available) B
• Provide point-and-click query for data sets according to instrument providing the data. A
• Obtain a subset of the data within a particular time interval (ideally including non-SIAM data). B
• Add customized raw plots (e.g., to QC more sophisticated systems). B

Corrections and modifications are encouraged, now or at any User Story meeting.

2004.04.20 Feedback

1. How to access, by instrument or?
2. Go with something more legible for non-specialist.
3. Operational diagnostics under one heading, science variables under another heading.
4. Minimize number of variables (don't show redundant ones).
5. Preconfigure list of most interesting variables.
6. There really should be a way to do our own groupings, not be limited to fixed layout.
7. A week is reasonable standard duration.
8. Would be really useful to have a contact responsible to guide the layout of the primary page.
9. Like to have date axis on top and bottom (or every so often).
10. How do we get data?
11. Confusing to get it from too many pages? Easy to access if there's a button on the page.
12. Would make more sense if redirected to a query page, allow changes to be made.
13. Provision for providing feedback about data would be valuable. (See Issues doc.)
14. Indicate data gaps. Could just plot points. Give user an option as to how to plot.
15. Would like to know reason for the gap.
16. Post-processing can help.
17. Maybe use sequence number.